Credit Card Auto-Fill

ABSTRACT

Differing embodiments of this disclosure may employ one or all of the several techniques described herein to perform credit card recognition using electronic devices with integrated cameras. According to some embodiments, the credit card recognition process may comprise: obtaining a first representation of a first image, wherein the first representation comprises a first plurality of pixels; identifying a first credit card region within the first representation; extracting a first plurality of sub-regions from within the identified first credit card region, wherein a first sub-region comprises a credit card number, wherein a second sub-region comprises an expiration date, and wherein a third sub-region comprises a card holder name; generating a predicted character sequence for the first, second, and third sub-regions; and validating the predicted character sequences for at least the first, second, and third sub-regions using various credit card-related heuristics, e.g., expected character sequence length, expected character sequence format, and checksums.

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure is related to the subject matter of commonly-assignedU.S. patent application Ser. No. ______, entitled, “Object-of-InterestDetection and Recognition with Split, Full-Resolution Image ProcessingPipeline,” Atty. Docket No. P22829US2 (119-0805U52), which was filed onMay 30, 2014 (“the '______ application) and commonly-assigned U.S.patent application Ser. No. ______, entitled, “Systems and Methods forCharacter Sequence Recognition with no Explicit Segmentation,” Atty.Docket No. P22963US1 (119-0807US1), which was filed on May 30, 2014(“the '______ application). The '______ application and '______application are each hereby incorporated by reference in theirentireties.

BACKGROUND

This disclosure relates generally to the field of image processing and,more particularly, to various techniques for perform credit cardrecognition using electronic devices with integrated cameras.

The advent of portable integrated computing devices has caused awide-spread proliferation of digital cameras. These integrated computingdevices commonly take the form of smartphones or tablets and typicallyinclude general purpose computers, cameras, sophisticated userinterfaces including touch-sensitive screens, and wirelesscommunications abilities through Wi-Fi, LTE, HSDPA and other cell-basedor wireless technologies. The wide proliferation of these integrateddevices provides opportunities to use the devices' capabilities toperform tasks that would otherwise require dedicated hardware andsoftware. For example, as noted above, integrated devices such assmartphones and tablets typically have one or two embedded cameras.These cameras comprise lens/camera hardware modules that may becontrolled through the general purpose computer using system softwareand/or downloadable software (e.g., “Apps”) and a user interfaceincluding, e.g., programmable buttons placed on the touch-sensitivescreen and/or “hands-free” controls such as voice controls.

One opportunity for using the features of an integrated device is tocapture and evaluate images. The devices' camera(s) allows the captureof one or more images, and the general purpose computer providesprocessing power to perform analysis. In addition, any analysis that isperformed for a network service computer can be facilitated bytransmitting the image data or other data to a service computer (e.g., aserver, a website, or other network-accessible computer) using thecommunications capabilities of the device.

These abilities of integrated devices allow for recreational, commercialand transactional uses of images and image analysis. For example, imagesmay be captured and analyzed to decipher information from the imagessuch as characters, symbols, and/or other objects of interest located inthe captured images. The characters, symbols, and/or other objects ofinterest may be transmitted over a network for any useful purpose suchas for use in a game, or a database, or as part of a transaction such asa credit card transaction. For these reasons and others, it is useful toenhance the abilities of these integrated devices and other devices fordeciphering information from images.

In particular, when trying to read a credit card with a camera, thereare multiple challenges that a user may face. Because of thewidely-varying distances that the credit card may be from the camerawhen the user is attempting to read the credit card, one particularchallenge is the difficulty in focusing the camera properly on thecredit card. Another challenge faced is associated with the difficultiesin reading characters with perspective correction, thus forcing the userto hold the card in a parallel plane to the camera to limit anypotential perspective distortions. One of the solutions to theseproblems available today is that the user has to be guided (e.g., viathe user interface on the device possessing the camera) to frame thecredit card (or other object-of-interest) in a precise location andorientation—usually very close to the camera—so that sufficient imagedetail may be obtained. This is challenging and often frustrating to theuser—and may even result in a more difficult and time-consuming userexperience than simply manually typing in the information of interestfrom the credit card. It would therefore be desirable to have a systemthat detects the credit card (or other object-of-interest) inthree-dimensional space, utilizing scaling and/or perspective correctionon the image, thus allowing the user more freedom in how the credit card(or other object-of-interest) may be held in relation to the cameraduring the detection process.

Another challenge often faced comes from the computational costs ofcredit card recognition (or other object-of-interest recognition)algorithms, which scale in complexity as the resolution of the cameraincreases. Therefore, in prior art implementations, the camera istypically running in a low resolution mode, which necessitates the closeframing of the card by the user in order for the camera to readsufficient details on the card for the recognition algorithm to worksuccessfully with sufficient regularity. However, placing the card insuch a close focus range also makes it more challenging for the camera'sautofocus functionality to handle the situation correctly. A finalshortcoming of prior art optical character recognition (OCR) techniques,such as those used in credit card recognition algorithms, is that theyrely on single-character classifiers, which require that the incomingcharacter sequence data be segmented before each individual charactermay be recognized—a requirement that is difficult—if not impossible—inthe credit card recognition context.

The inventors have realized new and non-obvious ways to make it easierfor the user's device to detect and/or recognize the credit card (orother object-of-interest) by overcoming one or more of theaforementioned challenges. As used herein, the term “detect” inreference to an object-of-interest refers to an algorithm's ability todetermine whether the object-of-interest is present in the scene;whereas the term “recognize” in reference to an object-of-interestrefers to an algorithm's ability to extract additional information froma detected object-of-interest in order to identify the detectedobject-of-interest from among the universe of potentialobjects-of-interest.

SUMMARY

Some images contain decipherable characters, symbols, or otherobjects-of-interest that users may desire to detect and/or recognize.For example, some systems may desire to recognize such characters and/orsymbols so that they can be directly accessed by a computer in aconvenient manner, such as in ASCII format. Some embodiments of thisdisclosure seek to enhance a computer's ability to detect and/orrecognize such objects-of-interest in order to gain direct access tocharacters or symbols visibly embodied in images. Further, by using anintegrated device, such as a smartphone, tablet or other computingdevice having an embedded camera(s), a user may capture an image, havethe image processed to decipher characters, and use the decipheredinformation in a transaction.

One example of using an integrated device as described above to detectand/or recognize an object-of-interest is to capture an image of anobject having a sequence of characters, such as a typical credit card,business card, receipt, menu, or sign. Some embodiments of thisdisclosure provide for a user initiating a process on an integrateddevice by activating an application or by choosing a feature within anapplication to begin a transaction. Upon this user prompt, the devicemay display a user interface that allows the user to initiate an imagecapture or that automatically initiates an image capture, with thesubject of the image being of an object having one or more sub-regionscomprising sequences of characters that the user wishes to detect, suchas the holder name, expiration date, and account number fields on atypical credit card. The sequences of characters may also be comprisedof raised or embossed characters, especially in the case of a typicalcredit card.

Differing embodiments of this disclosure may employ one or all of theseveral techniques described herein to perform credit card recognitionusing electronic devices with integrated cameras. According to someembodiments, the credit card recognition process may comprise: obtaininga first representation of a first image, wherein the firstrepresentation comprises a first plurality of pixels; identifying afirst credit card region within the first representation; extracting afirst plurality of sub-regions from within the identified first creditcard region, wherein a first sub-region comprises a credit card number,wherein a second sub-region comprises an expiration date, and wherein athird sub-region comprises a card holder name; generating a predictedcharacter sequence for the first, second, and third sub-regions; andvalidating the predicted character sequences for at least the first,second, and third sub-regions using various credit card-relatedheuristics, e.g., expected character sequence length, expected charactersequence format, and checksums.

Still other embodiments of this disclosure may employ one or all ofseveral techniques to use a “split” image processing pipeline that runsthe camera at its full resolution (also referred to herein as“high-resolution”), while feeding scaled-down and cropped versions ofthe capture image frames to a credit card recognition algorithm. (It isto be understood that, although the techniques described herein will bediscussed predominantly in the context of a credit card detector andrecognition algorithm, the split image processing pipeline techniquesdescribed herein could be applied equally to any otherobject-of-interest for which sufficient detection and/or recognitionheuristics may be identified and exploited, e.g., faces, weapons,business cards, human bodies, etc.) Thus, one part of the “split” imageprocessing pipeline described herein may run the credit card recognitionalgorithm on scaled down (also referred to herein as “low-resolution”)frames from the camera, wherein the scale is determined by the optimumperformance of that algorithm. Meanwhile, the second part of the “split”image processing pipeline may run a rectangle detector algorithm (orother object-of interest detector algorithm) with credit card-specificconstrains (or other object-of interest-specific constraints) in thebackground. If the rectangle detector finds a rectangle matching theexpected aspect ratio and minimum size of a credit card that can beread, then it may crop the card out of the “high-resolution” camerabuffer, perform a perspective correction, and/or scale the rectangle tothe desired size needed by the credit card recognition algorithm andthen send the scaled, high-resolution representation of the card to thedetection algorithm for further processing.

One reason for using the split image processing pipeline to operate onthe “high resolution” and “low resolution” representations of theobject-of-interest concurrently (rather than using solely the “full” or“high resolution” pipeline) is that there are known failure casesassociated with object-of-interest detector algorithms (e.g., rectangledetector algorithms). Examples of failure cases include: 1.) The userholding the credit card too close to the camera, resulting in some edgesbeing outside the frame. This may fail in the rectangle detector (i.e.,not enough edges located to be reliably identified as a valid rectangleshape) but work fine in the direct path of feeding the “low-resolution”version of the image directly to the credit card recognition engine. 2.)Some particular kinds of credit cards or lighting and backgroundscenarios will make it very difficult for the edge detector portion ofthe rectangle detector to reliably identify the boundaries of the creditcard. In this second case, the user would likely be instructed toattempt to frame the card very closely to the camera, so that the creditcard recognition engine alone can read the character sequences of thecard. In some embodiments, if no valid credit card has been found by therectangle detector after a predetermined amount of time, the userinterface (UI) on the device may be employed to “guide” the user toframe the card closely.

Advantages of this split image processing pipeline approach toobject-of-interest recognition include the ability of the user to holdthe card more freely when the camera is attempting to detect the cardand read the character sequences (as opposed to forcing the user to holdthe card at a particular distance, angle, orientation, etc.). Thetechniques described herein also give the user better ability to movethe credit card around in order to avoid specular reflections (e.g.,reflections off of holograms or other shiny card surfaces). In mostcases, the credit card will also be read earlier than in the prior artapproaches in use today.

Still other embodiments of this disclosure may be employed to performcharacter sequence recognition with no explicit character segmentation.According to some such embodiments, the character sequence recognitionprocess may comprise generating a predicted character sequence for afirst representation of a first image comprising a first plurality ofpixels by: sliding a well-trained single-character classifier, e.g., aConvolutional Neural Network (CNN), over the first representation of thefirst image one pixel position at a time until reaching an extent of thefirst representation of the first image in a first dimension (e.g.,image width); recording a likelihood value for each of k potentialoutput classes at each pixel position, wherein one of the k potentialoutput classes comprises a “background class”; determining a sequence ofmost likely output classes at each pixel position; decoding the sequenceby removing identical consecutive output class determinations andbackground class determinations from the determined sequence; andvalidating the decoded sequence using one or more predeterminedheuristics, such as credit card-related heuristics.

In still other embodiments, the techniques described herein may beimplemented as methods, encoded in instructions stored in non-transitoryprogram storage devices, or implemented in apparatuses and/or systems,such as electronic devices having cameras, memory, and/or programmablecontrol devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the output of a rectangle detector on an exemplaryimage comprising a representation of a credit card, in accordance withone embodiment.

FIG. 2 illustrates a single-path, low-resolution, object-of-interestrecognition image processing pipeline, in accordance with the prior art.

FIG. 3 illustrates an example of a cropped, perspective corrected, andscaled object-of-interest comprising a representation of a credit card,in accordance with one embodiment.

FIG. 4 illustrates a split path, high resolution, object-of-interestdetection and recognition image processing pipeline, in accordance withone embodiment.

FIG. 5 illustrates an exemplary rectangle detector process in flowchartform, in accordance with one embodiment.

FIG. 6 illustrates an exemplary credit card recognition process inflowchart form, in accordance with one embodiment.

FIG. 7 illustrates an exemplary convolutional neural network (CNN), inaccordance with one embodiment.

FIG. 8 illustrates an exemplary digit sequence in a natural image, inaccordance with one embodiment.

FIG. 9A illustrates an exemplary activation lattice using a pre-trainedCNN, in accordance with one embodiment.

FIG. 9B illustrates an exemplary activation lattice using an adapted CNNand a decoded character sequence, in accordance with one embodiment.

FIG. 10 illustrates an exemplary credit card recognition process using asliding CNN window in flowchart form, in accordance with one embodiment.

FIG. 11 illustrates a simplified functional block diagram of anillustrative electronic device, according to one embodiment.

DETAILED DESCRIPTION

Systems, methods and program storage devices are disclosed herein forperforming credit card recognition using electronic devices withintegrated cameras. The techniques disclosed herein are applicable toany number of electronic devices with displays and cameras, such as:digital cameras, digital video cameras, mobile phones, personal dataassistants (PDAs), portable music players, monitors, and, of course,desktop, laptop, and tablet computers.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the inventive concept. As part of this description,some of this disclosure's drawings represent structures and devices inblock diagram form in order to avoid obscuring the invention. In theinterest of clarity, not all features of an actual implementation aredescribed in this specification. Moreover, the language used in thisdisclosure has been principally selected for readability andinstructional purposes, and may not have been selected to delineate orcircumscribe the inventive subject matter, resort to the claims beingnecessary to determine such inventive subject matter. Reference in thisdisclosure to “one embodiment” or to “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one implementation of theinvention, and multiple references to “one embodiment” or “anembodiment” should not be understood as necessarily all referring to thesame embodiment.

It will be appreciated that, in the development of any actualimplementation (as in any development project), numerous decisions mustbe made to achieve the developers' specific goals (e.g., compliance withsystem- and business-related constraints), and that these goals may varyfrom one implementation to another. It will also be appreciated thatsuch development efforts might be complex and time-consuming, but wouldnevertheless be a routine undertaking for those of ordinary skill in thedesign of an implementation of image processing systems having thebenefit of this disclosure.

Referring now to FIG. 1, the output 110/115 of a rectangle detector onan exemplary image 100 comprising a representation of a credit card 105is illustrated, in accordance with one embodiment. As shown in FIG. 1,the credit card 105 that the system is attempting to detect (andpresumably to subsequently read the relevant, credit card-relatedinformation from) is being held in the hand of a user at a comfortabledistance from the camera, with no user interface guidelines orinstructions directing the user where to hold the credit card withrespect to the image frame. In the example shown in FIG. 1, credit card105 comprises three pieces of relevant, credit card-related informationthat a credit card recognition algorithm would likely want to read: thecredit card number 105A, the credit card expiration date 105B, and thecredit card holder name 105C. Various challenges associated withdetecting and recognizing character sequences in these three canonicalcredit card information fields will be discussed in further detailbelow. As is typical, the rectangle detector that has been run onexemplary image 100 has located two potential valid rectangles:rectangle 110 (shown in dashed-line form) and rectangle 115 (shown insolid-line form). The various components of a rectangle detector may beconfigured to select the best rectangle from among the located rectanglecandidates returned by the rectangle detector, as will be discussed infurther detail below.

Referring now to FIG. 2, a single-path, low-resolution,object-of-interest recognition image processing pipeline 200 isillustrated, in accordance with the prior art. In the exemplary imageprocessing pipeline 200, camera 205 returns an image 210 considered tobe “low-resolution.” A low-resolution image may comprise, for example, avideo frame having a 640 pixel by 480 pixel resolution. Thelow-resolution image 210 is then simply passed to an object recognitionengine 215. In the example shown in FIG. 2, the object-of-interest is acredit card, so the object recognition engine 215 may attempt toidentify and read the various information fields on the credit card,such as the credit card number, credit card expiration date, and creditcard holder name fields, as discussed above with reference to FIG. 1.

Because the algorithm to identify and read the various informationfields on the credit card can be very computationally expensive, in someimplementations, there is no computationally feasible choice other thanto use low-resolution images (e.g., 640 pixels by 480 pixels) forobject-of-interest recognition. Otherwise, there would be too many imagepixels to operate on and read the credit card information in real-timeoff the camera's video stream. Additionally, for most characterrecognition algorithms, there is a minimum height required for thealgorithm to be able to recognize the letters, so the credit card needsto be positioned fairly close to the camera for any implementationoperation on low-resolution image data. With the object-of-interestpositioned very close to the camera, i.e., in the macro-focus range, thecamera's lens moves very little, so the depth of field is very shallow.This makes it difficult for the camera to achieve proper focus. Thefarther away the object-of-interest is from the camera, the less thecamera has to move to achieve proper focus. As will be discussed below,this provides further motivation for the split image processing pipelineto be run concurrently in both low-resolution and high-resolution modes.

Referring now to FIG. 3, an example of a cropped, perspective corrected,and scaled object-of-interest comprising a representation of a creditcard is illustrated, in accordance with one embodiment. As discussedabove with reference to FIG. 1, image 100 has been analyzed, andpotential rectangles 110 and 115 have been detected by a rectangledetector tuned to detect typical credit card shapes. For the sake ofexample, rectangle 110 has been chosen in FIG. 3 as the best rectanglecandidate in image 100. Located rectangle 110 has been cropped fromimage 100, and each of its corners have been perspective corrected viaprocess 300, resulting in scaled, cropped, and perspective-correctedrepresentation 305 of credit card 105. As may now be seen more clearly,the various credit card information fields, i.e., credit card number305A, the credit card expiration date 305B, and the credit card holdername 305C, are now likewise scaled to larger heights, straightened intoa horizontal row of characters, and at a higher resolution (since theyhave been cropped from the full-resolution image frame 100. According tosome embodiments, the act of perspective correction may be performed bycalls to existing image processing code modules, such as a CORE IMAGEfilter, provided by APPLE INC. According to other embodiments, theresulting scaled, cropped, and perspective-corrected representation 305of credit card 105 may be scaled to the same size of the low-resolutionimage frame (e.g., 640 pixels by 480 pixels), so that theobject-of-interest recognition algorithm may be run identically on theframes provided by both the low-resolution and high-resolution paths ofthe split image processing pipeline. As may now be more clearlyunderstood, in the case of the high-resolution path, theobject-of-interest, i.e., credit card, will take up the entire extent ofthe 640 pixel by 480 pixel image frame, whereas, in the case of thelow-resolution path, the extent of the 640 pixel by 480 pixel imageframe taken up by the object-of-interest will be determined by howclosely the user was holding the object-of-interest to the camera at thetime of capture. (In some embodiments, it has been empiricallydetermined that the object-of-interest should comprise at least 20% ofthe extent of the larger dimension of the image frame to have arealistic probability of successful object-of-interest recognition.)Thus, it may be expected that the high-resolution path may lead tohigher quality recognition results in many scenarios, e.g., scenarios inwhich the user is not holding the object-of-interest very close to thecamera.

Referring now to FIG. 4, a split path, high resolution,object-of-interest detection and recognition image processing pipeline400 is illustrated, in accordance with one embodiment. In the exemplaryimage processing pipeline 400, camera 405 returns a full resolutionimage 410 considered to be “high-resolution.” A high-resolution imagemay comprise, for example, a video frame having a 3,264 pixel by 2,448pixel resolution (i.e., 8 megapixels) or a 2,592 pixel by 1,936 pixelresolution (i.e., 5 megapixels). The high-resolution image 410 is thenconcurrently processed on both the high-resolution and low-resolutionpaths of the split image processing pipeline. In the example shown inFIG. 4, the low-resolution path begins by scaling the full resolutionimage 410 to a more manageable low-resolution size, e.g., 640 pixels by480 pixels. In some embodiments, this scaling may be performed byhardware scaler 415, such as a memory-to-memory (i.e., “M2M”) scaler,resulting in low resolution image 420. In other embodiments, scaler 415may also be implemented in software or performed by a graphicsprocessing unit (GPU). Low-resolution image 420 may then be passed toobject recognition engine 215, which, as discussed above with referenceto FIG. 2, may attempt to identify and read the various informationfields on the credit card, such as the credit card number, credit cardexpiration date, and credit card holder name fields.

With respect to the high-resolution path, an object-of-interest detector425 may be run on the full resolution image 410. According to someembodiments, object-of-interest detector 425 may comprise a rectangledetector, as will be described in greater detail with reference to FIG.5. Once the most likely object-of-interest candidate has been detectedby object-of-interest detector 425, the high-resolution path may proceedto crop, perspective correct, and scale the detected object-of-interest(block 430), resulting in a well-aligned, appropriately-sized,high-resolution image 435, consisting of only the object-of-interestcropped out of the original full resolution image 410. High-resolutionimage 435 may then also be passed to object recognition engine 215 inorder to attempt to identify and read the various information fields onthe credit card, such as the credit card number, credit card expirationdate, and credit card holder name fields. Because the image data fromthe two paths of the split image processing pipeline may reach theobject recognition engine 215 at different times, according to someembodiments, the first image that is evaluated as having a qualitymetric exceeding a first quality threshold value may be selected to haveits recognized information returned to the requesting process.

According to some embodiments, the split image processing pipeline maybe implemented in an electronic device having a multi-core architecture.In particular, each of the pipelines may run on a different core.

Referring now to FIG. 5, an exemplary rectangle detector process 500 isillustrated in flowchart form, in accordance with one embodiment. First,the input image (502) is taken and scaled to an appropriate size (504).According to some embodiments, the image may be scaled down to only 256pixels by 256 pixels before performing edge detection. Detecting edgesat a lower resolution filters out noise from the image. Once the edgeshave been located, the image data can be scaled back up tofull-resolution so that the character recognition process is more likelyto be successful.

Next, the process 500 will compute a gradient image (506) and perform adesired edge detection algorithm (508). According to some embodiments, aCanny edge detection process is used, although this is not strictlynecessary. Next, the process 500 may find edge pairs that areapproximately orthogonal, i.e. nearly perpendicular to each other (510),and generate potential quadrilateral candidates. The potentialquadrilateral candidates may then be pruned by size, aspect-ratio, orwhatever other object-of-interest heuristics are known to the detectorprocess. The process finally considers the quadrilateral candidates inconjunction with the edge detection information to find areas of strongoverlap with image edges (512), which serves as a final check in theprocess's determination of the strongest quadrilateral candidates tooutput to the requesting process (514).

Many variants to the rectangle detector process described with referenceto FIG. 5 may be employed, and other detectors may also be employed forshapes other than rectangles, such as squares, circles, human faces,etc. In particular, according to some embodiments disclosed herein, therectangle detector is taking advantage of the known aspect ratio of thecredit card, while dealing with perspective distortion, noise,background objects, patterns on the credit cards themselves, motion blur(e.g., due to motion of the camera, the motion of the hand holding thecard, or both) and occlusion of edges (in the hand-held credit cardcase), etc.

Referring now to FIG. 6, an exemplary credit card recognition process600 is illustrated in flowchart form, in accordance with one embodiment.First, as described above, the process may receive a representation of acredit card (or other object-of-interest) from both the high-resolutionpath of the split image processing pipeline (i.e., the path where therectangle detector has been used to crop and scale only therepresentation of the credit card out from the high-resolution of thecaptured image) (Step 605), as well as from the low-resolution path ofthe split image processing pipeline (i.e., the path where the scaledpreview frame has been sent directly to the object-of-interestrecognition algorithm) (Step 610).

As the object-of-interest recognition algorithm is receiving imageframes concurrently from each path of the split path image processingpipeline (e.g., in different threads and/or on different cores), it willperform region extraction (Step 615) and string recognition techniques(Step 620) in real-time on each stream of incoming image frames andcompare the quality of the recognized objects-of-interest in theincoming images to established quality metrics in order to determinewhether an object-of-interest has been recognized with sufficientconfidence (Step 625). In some embodiments, determining whether theobject-of-interest has been recognized with sufficient confidencecomprises determining whether the quality metric exceeds a first qualitythreshold value. The region extraction (Step 615) and string recognition(Step 620) steps will be described in further detail below.

In some embodiments, determining whether there is an object-of-interestrepresentation present in the incoming image with sufficient confidencemay involve reliance on the object-of-interest recognition algorithm, aswell as other object-of-interest-related heuristics. For example, in thecase of credit cards, checksums may be used to validate that the processis getting back a valid card number from the recognition engine. Thechecksum, as provided by ISO/IEC-7811 Part 1, uses a set of mathematicalequations to involving each of the digits in the credit card number(other than the last digit) in order to set the last digit of the creditcard number. Thus, if any recognized digit in the credit card number iswrong, the checksum will not equal the correct number for the last digitof the credit card number. When the object-of-interest is a credit card,checks may also be done against the prefix of the credit card number todetermine whether the prefix represents a valid prefix for a majorcredit card vendor (e.g., American Express, MasterCard, VISA, etc.).Other high-level filtering heuristics may also be used, such as thepotential character classes the CNN or other single-character classifiershould recognize in the incoming image. In one embodiment, the onlyvalid character classes are the numbers 0-9 and a “background” class, aswill be described in further detail below. In the case of credit cardholder names, the characters A-Z may also be valid character classes.Because image backgrounds are often quite complex, numbers may beclipped incorrectly, e.g., a ‘9’ might appear to be a ‘1’ if it theregion around the credit card number field is extracted incorrectly. Ifan object-of-interest passes each of these object-of-interest-relatedconstraints, the process may have sufficient confidence that it hasdetected a valid object-of-interest and proceed to Step 650 to performstring clean up and validation and, finally, return the formatted andvalidated credit card data to the requesting process (Step 655).According to some embodiments, the credit card should be extracted at aresolution high enough that the credit card number, expiration date andcard holder name images can be extracted at minimum pixel height in afirst dimension, e.g., 28 pixels in height.

In some embodiments, the process 600 will use the first image framepassed to it that has a sufficient confidence score—whether it came fromthe high-resolution path or the low-resolution path. If, at Step 625, noobject-of-interest representation is recognized with sufficientconfidence after a first predetermined amount of time, t1, has passed(but before a second predetermined amount of time, t2, has passed,wherein t2>t1), the process may proceed to use the UI on the display ofthe camera-enabled device to guide the user's placement of the creditcard with respect to the camera in order to lead to a higher likelihoodof detection with sufficient confidence (Step 645). Once anobject-of-interest representation is recognized with sufficientconfidence, the process will proceed to Step 650 to perform string cleanup and validation. If no object-of-interest representation may berecognized with sufficient confidence after a second predeterminedamount of time, t2, has passed (Step 635)—even after using the UI toguide the user's placement of the credit card—the process may time outand exit (Step 640) and inform the user to try again later, perhapsunder different lighting conditions or against a different background.Additionally, or alternatively, a user may be informed of knownsuboptimal conditionals without requiring a timeout. For example,low-lighting conditions could be detected and reported to the userbefore a full timeout occurred.

Region Extraction (Step 615)

The credit card number region may be extracted from the incoming creditcard image based on the ISO/IEC-7811 Part 1 standard, which specifiesthe embossed regions of the credit card (Step 615). In one embodiment, afull cut of the credit card identification region is passed to the cardobject recognition engine 215, which will attempt to recognize theregion as a credit card number. The object recognition engine 215 maythen provide potential 15- and 16-digit results back to the process 600,which results may then be evaluated to determine whether they representa valid credit card number, e.g., using Luhn checksums, as well as aprefix verification that checks to ensure that the first digit(s) of thecredit card number are not outside the range of expected bankinginstitutions.

If a valid credit card number is found, further card regions may beexamined to attempt to find a valid card expiration date and card holdername. The second embossed region from ISO/IEC-7811 Part 1 specifies aname and address area. This area may be extracted, and a series of cutsmade based on a set of probable locations given from a variety ofgenuine cards. For example, expiration dates are expected to be in oneof two general formats: either day-month-and-year or justmonth-and-year. “Wide” and “narrow” regions may then be cut in theexpected date locations and passed to the object recognition engine 215.Due to the variability of the overall credit card cut itself, severalvertical offsets—as well as cuts of varying widths—may be made toattempt to cover cases where the date lies slightly above, below, orbeyond the expected regions. Once a valid date is found, it may besaved, and the extraction process may proceed to attempt to findcardholder name is made.

For the card holder name field, full lines from the address area arepassed to the engine also using half-line increments to handlecardholder name appearing in between image lines. Once a valid name isfound it is returned and results are returned to the user. If cardholdername or expiration date regions are not found, the system makes severalmore attempts through the whole pipeline to try to recover cardholdername and expiration date. If both are still not found whatever resultsare found on the final frame are returned to the user.

String Recognition (Step 620)

Once a region of interest containing a credit card number, an expirationdate or a cardholder name is isolated, the resulting image may be sentto the string recognition portion of the object recognition engine 215(Step 620). According to some embodiments, the object recognition engine215 takes an image as its input and returns a list of possible characterlabel sequences. As will be discussed in further detail below, thestring recognizer is designed to work without any a priori knowledge ofthe length of the label sequence, but, if known a priori, may also beused to produce a character label sequence of a given character length.

For each of the three fields, i.e., credit card number, expiration date,and cardholder name, an independent single character classifier may bepretrained before the classifier is put into use. According to someembodiments, a Convolutional Neural Network (CNN) with one output foreach symbol in the alphabet (plus an additional “background class”) isused for this task. Instead of trying to explicitly segment thecharacter string into individual characters and recognize potentialcharacter candidates one at a time, according to some embodimentsdescribed herein, the CNN classifier slides over the whole image, pixelby pixel, and the best-matching character sequence may be extracted fromthe resulting collection of activations. The resulting collection ofactivation probabilities at each pixel position in the image will alsobe referred to herein as the “activation lattice.” When creating theactivation lattice, the CNN recognizes the correct character class whenit is centered (or nearly-centered) over it, and predicts the“background class” when positioned over parts of the background imagefalling in between valid characters. As may now be more fullyappreciated, by utilizing the novel “background class” concept, thecharacter string may be recognized without performing explicit a priorisegmentation.

As will be discussed further with reference to FIGS. 9A and 9B, slidinga single-character CNN classifier over the input image may result in anoisy activation lattice, from which extracting the correct labelsequence may prove difficult and error-prone. The pre-trainedsingle-character recognizer may therefore be adapted over a training setconsisting of a collection of images with corresponding label sequences.Training the character recognizer may comprise changing the CNN'sparameters such that the predicted character string matches the labelsequence. This type of sequence training ensures that the optimizationcriterion is better aligned with the task at hand, which is to recognizea string of several characters. As compared to the pre-trained CNN,extracting the correct character label sequence from the trained CNN'sactivation lattice is more accurate and more robust.

As will be understood, the character classifiers may also be customizedfor the particular credit card information fields that they areoperating on:

Credit card number: The alphabet for the credit card number recognizermay consist of the ten digits (i.e., 0-9), and the string recognizer mayreturn two possible label sequences—one with 15 digits and one with 16digits (since both sequence lengths are supported by different creditcard vendors). Then, the potential credit card number sequence thatpasses the aforementioned checksum tests may be selected as the mostlikely credit card number character sequence.

Expiration date: The alphabet for the expiration date recognizer mayconsist of nineteen uppercase letters (i.e., those that are used in thevarious month abbreviations), ten digits (i.e., 0-9) and three specialcharacters (i.e., the period, dash, and forward slash). Becauseexpiration dates on credit cards have two common formats, i.e., those oflength five and those of length eight, the expiration date recognizermay return label sequences of both length five and length eight, withthe date sequence more strongly matching a tailored regular expressionsearch and/or an expected date format being selected as the most likelyexpiration date character sequence.

Card holder name: The alphabet for the card holder name recognizer mayconsist of twenty-six uppercase letters (i.e., A-Z), six specialcharacters (e.g., hyphens, periods, commas, forward slashes,apostrophes, and ampersands), and a space. Cardholder names have nofixed length, and the name recognizer therefore returns the most likelysequence for this task.

For all three tasks, training data may be extracted from annotatedcredit cards. For the single-character classifier, single characters andthe corresponding labels may be extracted. For the sequence trainingphase, images of the entire strings with the sequence labels arerequired.

String Clean Up and Validation (Step 650)

Signals returned from the object recognition engine 215 are often noisyand include additional or incorrect information, so to improve results,fields may be validated before being returned to the user (Step 650).

For example, expiration dates returned from the object recognitionengine 215 can appear in several different formats/styles: dd.mm.yy;dd/mm/yy; dd-mm-yy; mm/yy; mm.yy; mm-yy; and mm/yy. In some embodiments,the recognized expiration dates are only returned if they match, e.g.,by a regular expression search, one of these expected date formats.

Names often come back very close (but not exact) to the expected names,so, according to some embodiments, a post-processing step of searching auser's

“Address Book” application (or similar database directory of known,i.e., valid, contacts) may be employed in order to find the closestedit-distance match in the Address Book to the recognized card holdername string. In this context, valid character strings refer to stringsfor which there is a particular reason or confirmation from anauthoritative third party source that the string in question is, infact, a valid string for the relevant context (e.g., a name may bepre-validated by appearing in a user's Address Book application, and aword or sequence of characters may be identified as valid by virtue ofappearing in a language model of a language of interest). If the matchbetween the predicted card holder name string and the Address Book entryis sufficient close, some embodiments may replace the recognized cardholder name string with the closest match from the Address Book orsimilar application. Multiple checks may be made, as names appearing oncredit cards sometimes include middle names, prefixes (e.g., Mr., Mrs.,Dr., etc.), abbreviations, etc.—and sometimes they do not.

Some embodiments may additionally employ support for what will bereferred to herein as a “language model.” Utilizing such a languagemodel, the string validation process may analyze the distribution ofcharacters and leverage knowledge from the language model regarding howlikely certain characters are to follow other characters. Languagemodels may be established by first examining a large corpus of valid andrelevant names and then computing models, which may later be used toprovide a confidence measure as to whether a recognized string is or isnot likely a name—even if it's not in the user's Address Book.Incorporating the language model during the decoding phase maypotentially help the CNN classification engine recover from ambiguous orlow-confidence activations. Such incorporation may be done in variousways, e.g., lattice rescoring, simple score weighting, or moresophisticated integration into the recognition engine. Commonlinguistics techniques, such as those employed in handwriting/drawingrecognition engines may be employed to leverage a character'ssurrounding context in order to help disambiguate the true identity ofcharacters. Thus, the character recognition scores from the objectrecognition engine 215 may be intelligently combined with the languagemodel scores to enhance the string validation portion of the objectrecognition engine 215.

Convolutional Neural Networks (CNNs)

The ability of multi-layer neural networks trained with gradient descentto learn complex, high-dimensional, non-linear mappings from largecollections of examples make them good candidates for image recognitiontasks. A trainer classifier (normally, a standard, fully-connectedmulti-layer neural network can be used as a classifier) categorizes theresulting feature vectors into classes. However, it could have someproblems that may influence the character recognition results. Theconvolution neural network solves this shortcoming of traditionalclassifiers to achieve improved performance on pattern recognitiontasks.

The CNN is a special form of multi-layer neural network. Like othernetworks, CNNs are trained by back propagation algorithms. Thedifference is that the convolutional network combines threearchitectural ideas to ensure some degree of shift, scale, anddistortion invariance: local receptive field, shared weights (or weightreplication), and spatial or temporal sub-sampling. CNNs have beendesigned especially to recognize patterns directly from digital imageswith a minimum of pre-processing operations. The preprocessing andclassification modules are within a single integrated scheme.

A typical convolutional neural network may consist of a set of severallayers. The values of the feature maps for each layer are computed byconvolving the input layer with the respective kernel and applying anactivation function to get the results. Each convolution layer may befollowed by a sub-sampling layer, which reduces the dimension of therespective convolution layer's feature maps by a constant factor. Thelayers of the neural network may be viewed as a trainable featureextractor. Then, a trainable classifier may be added to the featureextractor, in the form of various fully-connected layers (i.e., auniversal classifier).

Referring now to FIG. 7, an exemplary convolutional neural network (CNN)700 is illustrated, in accordance with one embodiment. According to thisexemplary CNN, the model extracts simple feature maps at a higherresolution, and then converts them into more complex feature maps at acoarser resolution by sub-sampling a layer, e.g., by a factor of two.After two layers of convolution and subsampling, the resulting featuremap is too small for a third layer of convolution. Thus, the first twolayers of this neural network can be viewed as a trainable featureextractor. Then, a trainable classifier is added to the featureextractor, in the form of two fully-connected layers (i.e., a universalclassifier). Finally, the weights for each layer may be updated via theprocess of back propagation, which may begin with the last layer andmove backwards through the layers until the first layer is reached.

As shown in FIG. 7, input layer 705 comprises a ‘4’ character, e.g., asread from an exemplary credit card. The input layer 705 may undergoconvolution sub-sampling, resulting in a first plurality of smallerfeature maps 710. Each of these smaller feature maps 710 may undergo asecond round of convolution sub-sampling, resulting in a secondplurality of yet smaller feature maps 715. These features may then becombined with a trainable classifier and used as a universal classifier,i.e., a set of fully connected neurons 720. The universal classifier maybe used to generate an output layer 725 by classifying incomingcharacters into one of the potential output classes 730 (in oneembodiment, the output classes comprise 0-9) or a “background” class735. As will be understood, FIG. 7 is merely exemplary, and representsjust one embodiment of a possible CNN that may be used to classifyincoming characters. The various parameters and layers may be adjustedto fit a particular implementation.

Character Sequence Recognition With No Explicit Segmentation

In recent years, focus in research and industry has been on developingand employing powerful machine learning techniques that are applied tooptical character recognition (OCR) problems, where a grayscale image isassigned to one out of k predefined output classes. Many benchmarks aremost successfully solved with CNNs (and variants thereof) that use rawpixel intensities as their inputs.

A common shortcoming of such single-character classifiers is thatsequences need to be segmented before each individual character may berecognized. As a consequence, the success of such a sequence classifierrelies on good character segmentation. Using standard image processingtechniques (e.g., binarization and connected component analysis) onlyworks for images with a relatively uniform background. For OCR innatural images, often characterized by highly-varying backgrounds, it isalmost impossible to obtain a good segmentation. For these scenarios, asuccessful algorithm not only needs to classify segmented characters—butalso has to learn the segmentation. Various techniques have been used toattempt to solve this problem, e.g., over-segmentation, or usingrecurrent neural networks (RNNs) that learn to classify sequences frominput images. Both approaches have drawbacks, to which the inventorshave discovered novel and non-obvious solutions.

Thus, disclosed herein are systems and methods that adapt to varyingbackgrounds and varying character spacings without substantiallydegrading the classification accuracy of character sequences in naturalimages. Referring now to FIG. 8, an exemplary digit sequence in anatural image 800 is illustrated, in accordance with one embodiment. Inthis example, the image has the sequence “523” across several differentnoisy backgrounds and with several intervening non-character featuresinterspersed with the characters. As will be discussed below withrespect to FIGS. 9A and 9B, a sliding, pretrained CNN window may be usedto construct activation lattice(s) that may be “decoded” to extract acharacter sequence from the natural image without performingsegmentation.

Instead of explicitly trying to segment and recognize potentialcandidates, according to some embodiments described herein, a CNN slidesover the whole image, pixel-by-pixel, and the best matching charactersequence may be extracted from the resulting collection of activations,referred to herein as the “activation lattice.” Each column in thislattice (see, e.g., activation lattice 930 in FIG. 9A and activationlattice 975 in FIG. 9B) corresponds to the activations of a CNN centeredat this pixel in the input image (see, e.g., input image 900 in FIG. 9Aand input image 950 in FIG. 9B). The CNN outputs may then be normalized,e.g., with a softmax activation function, to be between 0 (white) and 1(black) (and sum to unity), and can be interpreted as posterior classprobabilities of the input image belonging to class k. Each rowcorresponds to the activations of the k^(th) class across the image. InFIGS. 9A and 9B, activations in each row 915 correspond to digits (0 to9 from top to bottom) and an additional background class, ‘g’ (bottomrow).

Sliding a pretrained digit classifier 905/955 over the input image(e.g., along the path of arrows 920/980 in FIGS. 9A and 9B) results in anoisy activation lattice (see, e.g., activation lattice 930 in FIG. 9A).Arrows 910/960 illustrate the correspondence between the position of theclassifiers 905/955 and their corresponding activations 925/965 in theactivation lattices 930/975, respectively.

Thus, as may now be better appreciated, obtaining the correct labelsequence “523” from this activation lattice may prove difficult anderror-prone. In particular, the labels “5” and “2” are likely to beextract successfully, but the label “3” is likely to be missed (asevidenced by the lack of a defined activation position under the “3”digit in activation lattice 930). Furthermore, due to relatively highactivations for different classes at various positions throughout theimage, an additional wrong label is very likely to be included in anyprediction derived from the activation lattice 930.

One goal of this process is to obtain an activation lattice from whichthe correct sequence is extracted consistently, with high accuracy, andwithout knowing the string length a priori. To this end, according tosome embodiments, the pretrained CNN may be trained over a “trainingset,” i.e., a collection of images with corresponding label sequences,and then back propagating the sequence errors through a ConnectionistTemporal Classification layer (CTC)—without ever having to segment thesequence explicitly.

As opposed to the pretrained CNN shown in FIG. 9A, extracting thecorrect label “523” from the trained CNN's activation lattice is moreaccurate and more robust (see activation lattice 975 in FIG. 9B).Furthermore, the trained CNN learned to predict the background class‘g,’ for all but the regions that coincide with the digits “5,” “2,” and“3”—making any explicit segmentation unnecessary.

Compared with prior art solutions, this approach benefits from alladvantages of CTC training. Furthermore, this approach results in gainedefficiencies—not only because a more efficient CNN is used instead ofnotoriously difficult to train RNNs, but also because the pretrained CNNremedies the slow convergence seen with conventional CTC training.

Turning now to a preferred embodiment of the CNN classification withoutexplicit segmentation process, a pretrained CNN with k+1 output classes,i.e., one output for each symbol in the alphabet plus an additional“background class,” is created. For the sake of explanation, it will beassumed that the image containing the sequence to be classified ishorizontally aligned, with its shorter, i.e., vertical, dimension equalto the height of the CNN's receptive field. As shown in FIGS. 9A and 9B,the height of the CNN's receptive field 905/955 is equal to the verticaldimension of the natural image with the “523” number sequence in it. Incases where a broader cut is made from the incoming image, the incomingimage may be scaled to ensure that its height is at the predeterminedfixed size (e.g., a height of 28 pixels and a width that covers fullimage). Scaling of the image is permissible because the CNN can be maderesilient to scaling issues as it is trained. Alternatively, if theimage is not scaled, the classifier could be shifted vertically, withthe activation likelihoods summed (or averaged) over the vertical extentof the image at each pixel position.

Sliding the pretrained CNN from left to right over the input image(e.g., along the path of arrows 920/980 in FIGS. 9A and 9B) andrecording the activations at every pixel position, p, results in theactivation lattice, y_(k) ^(p), i.e., the posterior class probability ofa window centered at pixel p belonging to class k. The conditionalprobability of any path σ of length P through the activation latticegiven an input image x is:

$\begin{matrix}{{p\left( \sigma \middle| x \right)} = {\prod\limits_{p = 0}^{P - 1}\; {y_{\sigma_{p}}^{p}.}}} & (1)\end{matrix}$

The conditional probability of any sequence s of length S≦P, given aninput image x is:

$\begin{matrix}{{{p\left( s \middle| x \right)} = {\sum\limits_{\sigma \in \Omega}\; {p\left( \sigma \middle| x \right)}}},} & (2)\end{matrix}$

where Ω is the set of all paths σ of length P that result in theidentical sequence s after removing repetitive labels and the backgroundclass. The goal, then, as in standard neural network training, is tomaximize equation 2 over a training set T={x_(i), s_(i)}. The adaptationof the pretrained CNN is then performed using stochastic gradientdescent may proceeds in the following way:

-   -   1. Randomly pick an image xi with the corresponding label        sequence xi from the training set T.    -   2. Compute the derivative of equation 2 with respect to the        network outputs y_(k) ^(p).    -   3. Back propagate the error signal through the network and        perform a weight update.    -   4. Repeat Steps 1-3 above until reaching convergence.        (Convergence is reached when any further change in the model        parameters will no longer meaningfully impact recognition        accuracy.)

Referring again to FIG. 9A, an exemplary activation lattice 930 using apre-trained CNN is illustrated, in accordance with one embodiment. FIG.9A depicts an activation lattice that would be created with a CNN thatrecognized the characters “5,” “2,” and “3.” The area in the activationlattice 930 corresponding to the “3” is a bit noisy because theclassifier may not have seen a “3” before. If the CNN were thenretrained, resulting in the trained CNN of FIG. 9B, the activationlattice 975 would be more likely to show the isolated “blobs”corresponding to the correct character classes, located at the positionsin the lattice corresponding to the positions of the characters in theimage.

Activation Lattice Decoding

Once the activation lattice has been created for a given input image, itmust be decoded to determine which characters (and how many characterstotal) are in the input image. Different heuristics have been developedby the inventors to find so-called “clusters” of activations within thelattice that may be segmented into a single character, e.g. a “3.” Oncea region has been located, the process may be iterated until the entiresequence has been traversed.

A naïve approach to activation lattice decoding may simply take thelargest activation(s) across the lattice only. However, according tosome embodiment disclosed herein, the character sequence as a whole maybe analyzed to determine the most likely final result. For example, itis known that valid credit card numbers will have either fifteen orsixteen digits, so, according to some embodiments, the activationenergies of consecutive blocks may be summed, and the fifteen (and/orsixteen) largest activation energies may be kept as the decoded fifteen(and/or sixteen)-digit credit card number sequence. [In someembodiments, both fifteen and sixteen digit sequences are checkedbecause it is not always known a priori which vendor's credit card isbeing read.] Other credit card-related heuristics may also be employed,such as the checksum and vendor-prefix heuristics described above, inorder to validate whether the recognized sequence of characters isvalid. Similar techniques may be employed with respect to expirationdates, which typically comprise sequences of five or eight characters.With the credit card holder names, the length of the sequence is notknown a priori, so different techniques may be employed, such asremoving consecutive repetitive activations and backgrounds characterclasses, as will be discussed in further detail below with reference toFIG. 9B.

Other credit card-related heuristics that may help with the decoding ofthe activation lattice include the fact that the fixed geometry ofembosser machines provides an “expected width” between digits. Forexample, if it is known that certain characters in the credit cardnumber sequence have center lines that are 2 mm apart, the decoding ofthe activation lattice may be biased towards strong activations (aswould be typical), with the additional requirement that successiveactivation are located 2 mm apart. This further heuristic may be used toreject certain cases where, e.g., the engine hasn't learned a particularcharacter well yet or where the engine still thinks a particularactivation is ambiguous.

Turning back to FIG. 9B, an exemplary a decoded character sequence 970is shown based on the activation lattice 975. As mentioned above, theCTC retraining process may involve taking the activation lattice output,looking at the likelihoods for each of the potential output classes ateach pixel position, and determining a sequence with a length equal tothe number of pixels in the image's width. Steps 1-4 in FIG. 9Billustrate the following of the exemplary decoding heuristics outlinedabove: 1) Repetitive positions are removed; and 2) each time abackground class is repeated, it is also removed. For example: at Step1, the status of the decoded sequence is:“______55______222______33______” (wherein underscores represent afinding of the “background class”). After removing repetitive positionsand background classes, at Step 2, the status of the decoded sequenceis: “_(—)5_(—)2_(—)3_.” After removing the background classes fromconsideration, at Step 3, the status of the decoded sequence is: “5 23.” Finally, at Step 4, after removing blank spaces, the decodedsequence is determined to be: “523.”

Referring now to FIG. 10, an exemplary credit card recognition processusing a sliding CNN window 1000 is summarized at a high-level andillustrated in flowchart form, in accordance with one embodiment. First,the process receives the image with the candidate characters forrecognition (Step 1005). Next, the neural network classifier may beplaced over the image (after appropriate scaling, if necessary) at astarting position (Step 1010). In some embodiments, the startingposition may be the far left of a mainly horizontal image (i.e., animage that is much wider than it is tall), the process may proceed bymoving the classifier in a rightward direction across the extent of theimage. Next, the process may record a likelihood value for each of kpotential output classes at the current position of the neural networkclassifier over the image (Step 1015). In some embodiments, one of the kpotential output classes comprises a “background class.” When thevarious likelihood values (also referred to herein as “activationstrengths”) have been recorded, the process may determine whether thereare further positions in the image for the neural network classifier tobe placed over (Step 1020). If there are further positions, the processmay slide the neural network classifier over the image by one position,e.g., by one pixel (Step 1025). The process may then proceed byrecording the likelihood values at each position across the extent ofthe image until there are no further positions in the image for theneural network classifier to be placed over (‘NO’ at Step 1020).

At step 1030, a single “activation lattice” for the image may be createdby aggregating all the likelihood values recorded from all the imagepositions over which the classifier has been evaluated. Next, theprocess may determine the sequence of most likely output classes foreach pixel position (Step 1035). Next, various decoding heuristics, suchas those described above, may be employed by the process to decode thesequence of output classes into a single string of output characterslikely to correspond to the characters in the input image (Step 1040). Afinal step may involve validating the decoded sequence usingpredetermined heuristics, such as expected sequence length, validatedstring values (e.g., names in an Address Book), known valid sequenceprefixes, known accepted string formats, etc. (Step 1045). Finally, thepredicted character sequence for the image may be returned to therequesting process (Step 1050).

Referring now to FIG. 11, a simplified functional block diagram of anillustrative electronic device 1100 is shown according to oneembodiment. Electronic device 1100 may include processor 1105, display1110, user interface 1115, graphics hardware 1120, device sensors 1125(e.g., proximity sensor/ambient light sensor, accelerometer and/orgyroscope), microphone 1130, audio codec(s) 1135, speaker(s) 1140,communications circuitry 1145, digital image capture unit 1150, videocodec(s) 1155, memory 1160, storage 1165, and communications bus 1170.Electronic device 1100 may be, for example, a personal digital assistant(PDA), personal music player, mobile telephone, or a notebook, laptop,or tablet computer system.

Processor 1105 may be any suitable programmable control device capableof executing instructions necessary to carry out or control theoperation of the many functions performed by device 1100 (e.g., such asthe processing of images in accordance with operations in any one ormore of the Figures). Processor 1105 may, for instance, drive display1110 and receive user input from user interface 1115 which can take avariety of forms, such as a button, keypad, dial, a click wheel,keyboard, display screen and/or a touch screen. Processor 1105 may be asystem-on-chip such as those found in mobile devices and include adedicated graphics processing unit (GPU). Processor 1105 may be based onreduced instruction-set computer (RISC) or complex instruction-setcomputer (CISC) architectures or any other suitable architecture and mayinclude one or more processing cores. Graphics hardware 1120 may bespecial purpose computational hardware for processing graphics and/orassisting processor 1105 process graphics information. In oneembodiment, graphics hardware 1120 may include one or more programmablegraphics processing units (GPUs).

Sensor and camera circuitry 1150 may capture still and video images thatmay be processed to generate images, at least in part, by video codec(s)1155 and/or processor 1105 and/or graphics hardware 1120, and/or adedicated image processing unit incorporated within circuitry 1150.Images so captured may be stored in memory 1160 and/or storage 1165.Memory 1160 may include one or more different types of media used byprocessor 1105, graphics hardware 1120, and image capture circuitry 1150to perform device functions. For example, memory 1160 may include memorycache, read-only memory (ROM), and/or random access memory (RAM).Storage 1165 may store media (e.g., audio, image and video files),computer program instructions or software, preference information,device profile information, and any other suitable data. Storage 1165may include one more non-transitory storage mediums including, forexample, magnetic disks (fixed, floppy, and removable) and tape, opticalmedia such as CD-ROMs and digital video disks (DVDs), and semiconductormemory devices such as Electrically Programmable Read-Only Memory(EPROM), and Electrically Erasable Programmable Read-Only Memory(EEPROM). Memory 1160 and storage 1165 may be used to retain computerprogram instructions or code organized into one or more modules andwritten in any desired computer programming language. When executed by,for example, processor 1105, such computer program code may implementone or more of the methods described herein.

It is to be understood that the above description is intended to beillustrative, and not restrictive. The material has been presented toenable any person skilled in the art to make and use the invention asclaimed and is provided in the context of particular embodiments,variations of which will be readily apparent to those skilled in the art(e.g., some of the disclosed embodiments may be used in combination witheach other). In addition, it will be understood that some of theoperations identified herein may be performed in different orders. Thescope of the invention therefore should be determined with reference tothe appended claims, along with the full scope of equivalents to whichsuch claims are entitled. In the appended claims, the terms “including”and “in which” are used as the plain-English equivalents of therespective terms “comprising” and “wherein.”

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to: obtain a first representation of a first image, wherein the first representation comprises a first plurality of pixels; identify a first credit card region within the first representation; extract a first plurality of sub-regions from within the identified first credit card region, wherein a first sub-region comprises a credit card number, wherein a second sub-region comprises an expiration date, and wherein a third sub-region comprises a card holder name; generate a predicted character sequence for the first, second, and third sub-regions; and validate the predicted character sequences for at least the first, second, and third sub-regions, wherein the instructions to validate the first sub-region comprise instructions to determine a sequence length of the predicted character sequence for the first sub-region and evaluate a checksum on the predicted character sequence for the first sub-region, wherein the instructions to validate the second sub-region comprise instructions to determine a sequence length of the predicted character sequence for the second sub-region and evaluate a format of the predicted character sequence for the second sub-region, and wherein the instructions to validate the third sub-region comprises instructions to compare the predicted character sequence for the third sub-region with a first plurality of valid character sequences.
 2. The non-transitory program storage device of claim 1, wherein the instructions to identify a first credit card region within the first representation further comprise instructions to execute a rectangle detector on the first representation of the first image.
 3. The non-transitory program storage device of claim 2, wherein the rectangle detector comprises at least one of: aspect-ratio constraints and size constraints.
 4. The non-transitory program storage device of claim 1, wherein the instructions to identify a first credit card region within the first representation further comprise instructions to guide placement of the object-of-interest within the first image.
 5. The non-transitory program storage device of claim 1, wherein the instructions to extract a first plurality of sub-regions from within the identified first credit card region further comprise instructions to extract a plurality of cuts for each of the first plurality of sub-regions, when each of the plurality of cuts for each of the first plurality of sub-regions has a different length in at least one dimension from the other cuts for the respective sub-region.
 6. The non-transitory program storage device of claim 1, wherein the instructions to validate the first sub-region further comprise instructions to determine a sequence length of fifteen or sixteen characters for the predicted character sequence for the first sub-region.
 7. The non-transitory program storage device of claim 1, wherein at least one of the valid character sequences comprises at least one of the following: a name from an address book; and a character sequence from a language model.
 8. A system, comprising: a memory having, stored therein, computer program code; a digital camera; and one or more processing units operatively coupled to the digital camera and memory and configured to execute instructions in the computer program code that cause the one or more processing units to: obtain a first representation of a first image from the digital camera, wherein the first representation comprises a first plurality of pixels; identify a first credit card region within the first representation; extract a first plurality of sub-regions from within the identified first credit card region, wherein a first sub-region comprises a credit card number, wherein a second sub-region comprises an expiration date, and wherein a third sub-region comprises a card holder name; generate a predicted character sequence for the first, second, and third sub-regions; and validate the predicted character sequences for at least the first, second, and third sub-regions, wherein the instructions to validate the first sub-region comprise instructions to determine a sequence length of the predicted character sequence for the first sub-region and evaluate a checksum on the predicted character sequence for the first sub-region, wherein the instructions to validate the second sub-region comprise instructions to determine a sequence length of the predicted character sequence for the second sub-region and evaluate a format of the predicted character sequence for the second sub-region, and wherein the instructions to validate the third sub-region comprises instructions to compare the predicted character sequence for the third sub-region with a first plurality of valid character sequences.
 9. The system of claim 8, wherein the instructions to identify a first credit card region within the first representation further comprise instructions to execute a rectangle detector on the first representation of the first image.
 10. The system of claim 9, wherein the rectangle detector comprises at least one of: aspect-ratio constraints and size constraints.
 11. The system of claim 8, wherein the instructions to identify a first credit card region within the first representation further comprise instructions to guide placement of the object-of-interest within the first image.
 12. The system of claim 8, wherein the instructions to extract a first plurality of sub-regions from within the identified first credit card region further comprise instructions to extract a plurality of cuts for each of the first plurality of sub-regions, when each of the plurality of cuts for each of the first plurality of sub-regions has a different length in at least one dimension from the other cuts for the respective sub-region.
 13. The system of claim 8, wherein the instructions to validate the first sub-region further comprise instructions to determine a sequence length of fifteen or sixteen characters for the predicted character sequence for the first sub-region.
 14. The system of claim 8, wherein at least one of the valid character sequences comprises at least one of the following: a name from an address book; and a character sequence from a language model.
 15. A computer-implemented method, comprising: obtaining a first representation of a first image from a first digital camera, wherein the first representation comprises a first plurality of pixels; identifying, using a computer, a first credit card region within the first representation; extracting, using a computer, a first plurality of sub-regions from within the identified first credit card region, wherein a first sub-region comprises a credit card number, wherein a second sub-region comprises an expiration date, and wherein a third sub-region comprises a card holder name; generating, using a computer, a predicted character sequence for the first, second, and third sub-regions; and validating, using a computer, the predicted character sequences for at least the first, second, and third sub-regions, wherein validating the first sub-region comprises determining a sequence length of the predicted character sequence for the first sub-region and evaluating a checksum on the predicted character sequence for the first sub-region, wherein validating the second sub-region comprises determining a sequence length of the predicted character sequence for the second sub-region and evaluating a format of the predicted character sequence for the second sub-region, and wherein validating the third sub-region comprises comparing the predicted character sequence for the third sub-region with a first plurality of valid character sequences.
 16. The computer-implemented method of claim 15, wherein the act of identifying a first credit card region within the first representation further comprises executing a rectangle detector on the first representation of the first image.
 17. The computer-implemented method of claim 16, wherein the rectangle detector comprises at least one of: aspect-ratio constraints and size constraints.
 18. The computer-implemented method of claim 15, wherein the act of identifying a first credit card region within the first representation further comprises guiding, using a computer, placement of the object-of-interest within the first image.
 19. The computer-implemented method of claim 15, wherein the act of extracting a first plurality of sub-regions from within the identified first credit card region further comprises extracting a plurality of cuts for each of the first plurality of sub-regions, when each of the plurality of cuts for each of the first plurality of sub-regions has a different length in at least one dimension from the other cuts for the respective sub-region.
 20. The computer-implemented method of claim 15, wherein at least one of the valid character sequences comprises at least one of the following: a name from an address book; and a character sequence from a language model. 